Time shifting by concurrently recording and playing a data stream

ABSTRACT

A video stream may include portions that are stored while other portions are being displayed. In one embodiment, a portion of the video stream may be stored in digital storage media at one instance while in the next instance another portion of the stream is being read out of the storage media.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a divisional of U.S. patent application Ser. No. 09/343,870,filed Jun. 30, 1999, now U.S. Pat. No. 6,591,058, which is acontinuation-in-part of U.S. patent application Ser. No. 09/150,577,filed Sep. 10, 1998, now abandoned which is a continuation-in-part ofU.S. patent application Ser. No. 08/996,535, filed Dec. 23, 1997, whichissued on Nov. 12, 2002 as U.S. Pat. No. 6,480,667.

BACKGROUND

This present invention relates to the recording and playing back of avideo stream. A video stream includes any combination of audio and/orvideo data streams.

Video streams have typically been recorded on analog media such as avideo cassette. A video cassette recorder (VCR) is used to record thevideo stream on the video cassette. The video stream may come via abroadcast signal, via cable, via satellite signal, or from another videoplayback device. Once the video stream has been recorded, the VCR isused to rewind the recording medium and play what was recorded. However,due to the nature of the analog medium, once the VCR has startedrecording, it is not possible to play back the portion of the videostream that has already been recorded until the recording session isterminated.

Thus, there is a need for an improved way of recording portions of avideo transmission.

SUMMARY

In accordance with one embodiment, a method of displaying and recordinga video transmission includes requesting scene change time information.The scene change time information is used to identify scenes in storedvideo information.

In accordance with another embodiment, a method of detecting scenechanges in video and audio information includes using a first techniqueto identify a scene change. Audio information is used to assist inidentifying a scene change.

In accordance with another embodiment, a method of detecting scenechanges in compressed video information includes identifying scenechange information developed by the algorithm used to compress saidvideo information. That scene change information is used to identifyscene changes. The location of the scene change is marked to facilitateidentifying the video frame where the scene change occurs.

In accordance with another embodiment, a method for identifying whetherdigital or analog video information has been received includesattempting to demodulate the information using a digital technique.Based on the results of the demodulation attempt, the format of theinformation is determined.

In accordance with another embodiment, a method of storing and playingdigital video information includes receiving a plurality of programs ona single channel. Each of said programs is stored in association with anidentifier. A first program is played and then stopped. The identifierand the point when the play is stopped is stored. A second program isplayed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a video record and playback system inaccordance with one embodiment of the invention;

FIG. 2 is a flow chart of one embodiment of the method of providing atime-shifted video stream in accordance with the invention;

FIG. 3 shows one embodiment of performing block 206 of FIG. 2;

FIG. 4 shows one embodiment of an apparatus for storing the video streamon a hard disk in accordance with the invention;

FIG. 5 shows an embodiment of a method of using the storage unit as atemporary buffer in accordance with the invention;

FIG. 6 is a flowchart of the playback of a video stream catching up tothe incoming video stream for one embodiment of the invention;

FIG. 7 shows another embodiment, in which the user is able to suspendthe display of the incoming video stream in accordance with theinvention;

FIG. 8 shows an example of an embodiment of a display screen withseveral image frames taken at different times in accordance with theinvention;

FIG. 9 is a flow chart showing one embodiment of a system for recordingand playing back a video stream in accordance with the invention;

FIG. 10 is a block diagram of a computer system useful in implementingone embodiment of the present invention;

FIG. 11 is an exemplary TV display for implementing an embodiment of thepresent invention;

FIG. 12 is a flow chart showing the operation of another embodiment ofthe present invention;

FIG. 12A is a continuation of the flow chart of FIG. 12;

FIG. 13 is a top plan view of a remote control useful in one embodimentof the present invention;

FIG. 14 is a block diagram showing how data is transferred to and from amemory device in accordance with one embodiment of the presentinvention;

FIG. 15 is a flow chart showing the flow of input video information intoa storage device in accordance with one embodiment of the presentinvention;

FIG. 16 is a flow chart showing the flow of data from the storage devicein accordance with one embodiment of the present invention;

FIG. 17 shows a display with an electronic program guide in accordancewith one embodiment of the present invention;

FIG. 18 is a block diagram of a video transmission system in accordancewith one embodiment of the present invention;

FIG. 19 is a depiction of a video file associated with a given broadcastchannel;

FIG. 20 is a depiction of a request to a scene change detector for scenechange information;

FIG. 21 is a depiction of a response to a request for scene changeinformation;

FIG. 22 is a depiction of software that may be used by the receivershown in the system of FIG. 18;

FIG. 23 is a flow chart for a scene change detector in accordance withone embodiment of the present invention;

FIG. 24 is a graphical user interface that may be used in connectionwith a scene change detector;

FIG. 25 is a flow chart for detecting whether information received froma video transmitter is in a digital or analog format and for processingthat information for storage;

FIG. 26 is a depiction of a channel in a digital video distributionsystem;

FIG. 27 is a flow chart for allowing an intrachannel jump betweenprograms on a single channel in a digital video distribution system;

FIG. 28 is a schematic depiction of the relationship between a frame anda DC image; and

FIG. 29 is a flow chart for software for handling detecting scenechanges without full frame decompression.

DETAILED DESCRIPTION

FIG. 1 shows a block diagram of a video record and playback system 100in accordance with one embodiment of the invention. A video stream isreceived at the video input port 102. The video stream may be providedby a camera, a television signal, broadcast, cable, or satellitesignals, or another video playback device. In one embodiment, ananalog-to-digital conversion may be performed on an analog video streamto form a digital video bit stream. In a different embodiment, the videois already in digital form. The video record and playback system 100 maybe part of a system, such as a computer system or set top box, such thatthe video input port 102 may be part of a video capture card in thecomputer system or set top box.

The digital video stream from the video input port 102 is optionallycompressed at compression unit 104. In one embodiment, the video isalready compressed and no further compression is needed. The videostream is then stored in the storage unit 108. A buffer unit 106 may beused as temporary storage for providing larger sequential blocks ofvideo data to the storage unit 108. In one embodiment, the buffer unit106 comprises a random access memory that allows relatively quick accessto any portion of the stored video stream.

The video stream is played back by reading the video stream from thestorage unit 108. If the video stream was compressed in compression unit104, then a decompression unit 110 decompresses the retrieved videostream. The video stream is provided from a video output port 120, to amonitor or other display device such as a TV to provide sound and/orvideo to a user.

A removable storage unit 122 may also be included in video record andplayback system 100, Examples of removable storage units include awriteable compact disk read only memory (CD-ROM), writeable digitalvideo disk (DVD), a flash memory, or another hard disk. The availabilityof a removable storage unit 122 allows a user to transfer a recording ofa video stream stored in storage unit 108 to the removable storage unit122 and then to transfer the unit 122 to another system at a differentlocation.

In one embodiment, a processor 130 controls the operations of the videorecord and playback system 100. The compression unit 104 anddecompression unit 110 may be implemented in hardware, or thecompression and decompression functions of units 104 and 110 may beperformed by the processor 130. Processor 130 receives instructions fromfirmware/memory 140, using technology that is well known.

FIG. 2 shows a flow chart of an embodiment of a method of providing atime-shifted video stream in accordance with the present invention. Theflow chart begins at block 200, and continues at block 202 where thevideo stream is received. The recording of the video stream begins atblock 204. At block 206, playback of the recorded video stream isperformed by retrieving a portion of the video stream from the randomaccess storage unit while the recording of the incoming video streamcontinues. The retrieved portion of the video stream may be time-shiftedfrom the incoming video stream by a time delay. At block 208, theportion of the video stream retrieved from the random access storageunit is retrieved for display by a television or other display device.

In this way, the record and playback functions are decoupled from oneanother. The user may now begin watching a recorded TV show from thebeginning, e.g., prior to the show being completely recorded.

FIG. 3 shows an embodiment of a method for retrieving a portion of thevideo stream from the random access storage unit while continuing torecord the incoming video stream in accordance with the invention,although the invention is not restricted to this embodiment. In thedisclosed embodiment, the substantially simultaneous recording andplayback of the video stream is performed by multiplexing or alternatelystoring the video stream to the random access storage unit and readingof the video stream from the random access storage unit. The multiplexedor alternated stores and reads may occur quickly enough that the userdoes not notice an appreciable delay in the playback of the videostream, and the incoming video stream is not lost, e.g., all of thevideo stream is recorded. Thus, the record and playback aresubstantially simultaneous from the user's point of view.

In one embodiment, the random access storage unit comprises a hard disk.The retrieval of the time-shifted video signal from the hard disk isperformed at a first physical location (or sector) of the hard disk, andthe storing to the hard disk of the incoming video stream is performedat a different physical location (or sector) on the hard disk. Becauseit takes more time to jump back and forth between different sectors ofthe hard disk than to read and write to sequential locations in the samesector, data may be buffered to reduce the number of accesses to andfrom the hard disk using buffer 106. This increases the amount of datatransferred per access.

Additionally, because of time constraints for reading and writing to thehard disk, data may be compressed and decompressed to speed transfers toand from the hard disk.

In block 302, the video stream is stored in a random access storageunit. The video stream is optionally compressed and/or buffered prior tostorage. In block 304, the video stream is retrieved from the randomaccess storage unit. Buffering and/or decompression may be performedbefore providing the retrieved video stream to the video output port120.

The next portion of the video stream is stored (block 306) as describedin block 302. At block 308, the next portion of the video stream isretrieved as described in block 304. This process is repeated untileither the recording or playback cycle is terminated.

FIG. 4 shows one embodiment of an apparatus for storing the video streamin the storage unit 108 in accordance with the invention. Again, theinvention is not restricted in scope to the illustrated embodiments. Inthis embodiment, the video stream is stored as separate files 001 and009 on a hard disk, for example. The processor 130 keeps track of thefile and offset into the file of the data being played back, as well asthe file and offset into the file of the data being recorded. If therandom access storage unit is fast enough, more than one video streammay be recorded and played back at the same time.

Due to the nature of the random access storage unit being capable ofeasily recording over itself, the random access storage unit may act asa temporary buffer for recording the latest portion, or X number ofminutes, of an incoming video stream, where X is set up based upon thesize of the storage unit in this particular embodiment. In oneembodiment, X could be set up to be the entire storage unit. As newerportions of the video stream are received, they overwrite the olderportions of the video stream saved in the random access storage unit. Inthis manner, the temporary buffering of the video stream acts as acircular buffer. In one embodiment, the processor 130 maintains pointersto the beginning and ending points of the temporary buffer. Theprocessor 130 reassigns the pointers as newer portions of the videostream are received and/or older portions of the video stream areoverwritten.

FIG. 5 shows a flow chart 500 of an embodiment of one method for using atemporary buffer in accordance with the invention. At block 502, thevideo stream is received at an input. Recording of the video stream tothe storage unit begins at block 504. At block 506, older portions ofthe video stream are deleted as newer portions of the video stream arestored to the storage unit.

A user may initiate a playback cycle following block 506. For example,this may occur when the user wishes to re-view a video clip that he justsaw. In one embodiment, the user stops recording to the temporary bufferand plays back the last portion of the temporary buffer.

However, it may be more desirable to the user to be able to continuerecording as shown at block 508. A record and playback cycle (asdescribed with respect to FIG. 2) is started, in which the incomingvideo stream is recorded while the user re-views the last portion of thetemporary buffer. In this manner, after re-viewing the desired videoclip, the user can resume sequentially watching the video stream fromthe current point of the incoming video for substantially simultaneousplayback and record.

At block 510, after the record and playback cycle is completed, all orpart of the temporary buffer may be saved. Since the temporary bufferstores the latest X minutes of the video stream prior to the record andplayback cycle, all or part of the temporary buffer may be allocated tothe portion of the video stream saved during the record and playbackcycle. Other portions of the video stream may then be deleted from thestorage unit, or they may be marked as overwriteable and used as a newtemporary buffer.

FIG. 6 illustrates an embodiment of a method for playing back a videostream to allow the playback to catch up to the incoming video stream inaccordance with the invention. If the playback is performed at anoverall rate faster than the rate at which the incoming video stream isreceived, then the playback may catch up to the incoming video stream.

For example, playback of the video stream may have an overall ratefaster than the rate of the incoming video stream if the playback isfast forwarded, or if segments of the playback are skipped altogether.When the time delay of the time-shifted video stream being played backfalls below a certain threshold, the video and playback system 100 willcease providing the time-shifted video stream from the storage unit.Instead, the incoming video stream will be provided to the video outputport 120 directly. In one embodiment, a bypass 142, as shown in FIG. 1,allows the incoming video stream to be provided to the video output port120 directly.

When this happens, the user has caught up to the “live” broadcast, e.g.,the incoming video stream in this embodiment. The user may terminate therecording cycle, if he wishes. Alternatively, the user may put the videorecord and playback system 100 back into the temporary buffering mode inwhich only the latest portion of the video stream is stored.

FIG. 7 shows an embodiment in which the user is able to suspend thedisplay of the incoming video stream. This can be used for example, whenthe user is interrupted, and wishes to continue viewing the video streamafter the interruption. When interrupted, the user signals to the videostream and playback system 100 to suspend the display of incoming videostream. This can be done via a remote control, for example. At block702, the video output signal is suspended. In one embodiment, the videooutput port continues to provide a still image of the image present atthe instance when the suspend was encountered.

At block 704 the incoming video stream is recorded but is not displayedto the monitor. Instead the playback is paused at the point at which theuser indicated that the incoming video stream be suspended. When theuser is ready to view the video stream again, he can signal the videorecord and playback system 100 to un-suspend the video stream so that itplays back from the point at which the video stream was suspended, asshown in block 706.

The user may then view the video stream time shifted by the amount oftime that he suspended the incoming video stream, or he may fast forward(or rewind) through the time-shifted video stream. When playback of thetime-shifted video stream catches up to the point at which the incomingvideo stream is being recorded, the record and playback system 100 maydisplay the incoming video stream directly from incoming video streamwithout retrieving the video stream from the storage unit, as describedwith respect to FIG. 6. The recording of the video stream may then beterminated by the user, if desired.

Employing a random access storage unit for storage of the video streamfacilitates jumping to various points within the video stream. One wayof jumping is by retrieving different time slices of the video stream.For example, an image frame from the video stream can be retrieved fromthe storage unit at 1 minute intervals from a current position of thevideo stream. In one embodiment, an image frame at the current playbackposition +1 minute, current playback position+2 minutes, and so forthare retrieved and displayed on the TV screen in a mosaic pattern.

FIG. 8 shows an example of a display screen 800 which displays severalimage frames taken from the video stream at different times. In FIG. 8,the current playback position is designated as X. In one embodiment, thetime interval, T, is user programmable. The intervals may be multiplesof the time interval, as shown. A small interval may be used if the userwishes to skip a commercial, which usually lasts only a few minutes.Longer intervals such as a half hour may be useful for determining whichmovies are recorded on a storage unit.

After the image frames are displayed, the user is able to select one ofthe frames as a new starting point at which to begin an operation, suchas a playback or record operation. By using such a story boardingmethod, it is easy for a user to quickly jump to a desired locationwithin the video stream.

In one embodiment, the record and playback system 100 is able to detecta black screen or fade out, such as those which accompany the beginningor end of a commercial. This is useful in editing a video stream. Thesystem may also record a series of cue points (e.g., fade outs and fadeto black) for later reference.

Having the video stream stored on a random access storage unit such as ahard disk allows for easy editing of the video stream. Individual framesof the video stream may be deleted or replaced. For example, acommercial may be replaced by a fade-to-black sequence.

Once the video stream on the storage unit has been edited, it can bestored to a more permanent medium, such as a writeable CD-ROM, flashmemory, or another hard disk via the removable storage unit 122.

Referring now to the embodiment shown in FIG. 9, a flow chart 900 fordigitally recording a video stream in accordance with an embodiment ofthe invention begins by capturing the video stream as indicated in block902. If the stream is an analog stream, it may be digitized in ananalog-to-digital conversion process as indicated at block 904. Next thedigital stream may be encoded and compressed, for example using theMPEG2 compression scheme, as indicated in block 906. The stream isalternately read, as indicated at block 910, and stored, as indicated inblock 908, in a conventional storage device such as a hard disk drive, adigital video disk or a flash memory. Data that is read from the storagedevice may be decoded and decompressed using conventional technology,for example, as indicated in block 912, for display as indicated inblock 914.

A system 1000 in accordance with one embodiment of the presentinvention, shown in FIG. 10, includes a processor 1002. In oneembodiment, the processor may be coupled to an accelerated graphics port(AGP) chipset 1004 for implementing an accelerated graphics portembodiment. The chipset 1004 communicates with the AGP port 1005 and thegraphics accelerator 1006. The television 1010 may be coupled to thevideo output of the graphics accelerator 1006. The chipset 1004accommodates the system memory 1008. The chipset 1004 is also coupled toa bus 1012 which may be, for example, a peripheral componentinterconnect (PCI) bus (PCI Local Bus Specification, Version 2.1 Jun. 1,1995). The bus 1012 couples to TV tuner/capture card 1014 which iscoupled to an antenna 1015 or other video input port, such as a cableinput port, a satellite receiver/antenna or the like. The TV tuner andcapture card 1014 selects a desired television channel and also performsthe video capture function (block 902, FIG. 9).

One example of video capture card is the ISVR-III video capture cardavailable from Intel Corporation.

The bus 1012 is also coupled to a bridge 1016 which couples a hard diskdrive 1018. The software 1020, 1022, 1024, 1026, 1028, 1030, 2200, 2300,2500 and 2700 may be stored on the hard disk 1018. The bridge 1016 isalso coupled to another bus 1032. The bus 1032 may be coupled to aserial input/output (SIO) device 1034. The device 1034 is in turncoupled to conventional components such as a mouse 1036, a keyboard1038, and an infrared interface 1040. Also coupled to the bus 1032 is abasic input/output system (BIOS) 1042.

An example of a display 1100, shown in FIG. 11, for the television 1010may display a conventional television picture or program 1101 and, inaddition, may have superimposed over the screen, in a discrete fashion,a mouse selectable series of icons, such as the icons 1102 through 1114.When the mouse cursor 1116 selects the appropriate one of the icons, acorresponding feature may be implemented. Among the features that may beimplemented in this fashion include a pause or stop function 1102, astart record function 1104, a fast forward function 1106, a rewindfunction 1108, and a 10 second replay function 1110 which winds back 10seconds (or some other amount) and replays, and a catchup function 1112that begins playing back recorded content at a faster than normal rateuntil the display catches up with a live broadcast and an electronicprogram guide 1114. The above described functions 1102, 1104, 1106,1108, 1110, and 1112 may also be individual buttons on a remote controlunit.

Referring next to FIG. 12, a program 1020 called display is used tocontrol various features implemented by the computer 1000. Initially acheck is made at diamond 1200 to determine if an electronic programmingguide (EPG) is implemented. An electronic programming guide is anelectronic depiction of the various programs that are available atdifferent times. The electronic programming guide can be provided on adisk, over a modem, by an Internet connection. An electronic programmingguide can also be implemented using an interactive broadcastingtransmission such as Intercast® 2.0 interactive broadcasting softwareavailable from Intel Corporation although the invention is not limitedin scope in this respect.

If a particular television program is selected (even if the program isonly selected for viewing) on the EPG, an identifier for that particularprogram may be stored, in one embodiment, as indicated at block 1202,and automatic recording of the program begins (block 1204). In this waythe stored version of the program is available to implement the variousfeatures described herein even if the storage function was notparticularly requested. Because the information is stored in anoverwriting fashion, no harm is done by recording the information evenif the information turns out not to be useful.

Referring to FIG. 17, an electronic program guide user interface 1700may be deployed on the television 1010. When the user mouse clicks a boximage 1702, representing a given television program, using the cursor1116, that program is automatically recorded.

Continuing in FIG. 12, an inquiry (diamond 1206) checks whether the userhas selected the zoom feature for playback. If the user presses a zoombutton during playback, a zoom feature is implemented in one embodiment.In one embodiment, five predefined quadrants in the television displaymay be defined including four quadrants located peripherally around acentral quadrant. When the user selects the zoom feature the userindicates the appropriate quadrant for the zoom (block 1208). Theselected quadrant is scaled (block 1210) to produce a larger (zoom)picture.

Referring to block 1208, the pointer focus is identified to determinewhich quadrant will be zoomed. That quadrant is then scaled as indicatedat block 1210. Because the television program is continually beingstored, the scaling can be implemented without any loss of continuity ofthe television picture. After the user has completed looking at thezoomed depiction, the user can return to the program at the point wherehe or she departed to view the zoomed depiction. Thereafter, thedisplayed depiction viewed by the user may be time delayed from theactually broadcast program. However the user is able to view the programin its entirety even if slightly time delayed.

Next, a query is made at diamond 1212 to determine whether a pausefunction has been selected. If so, the playback is stopped as indicatedat block 1214 but recording continues unabated as shown in block 1216.

The next check determines whether the play feature has been selected(block 1218). If so the recorded content is played thereby terminatingthe paused display. However, while the recorded content is displayed,recording continues as indicated in block 1222 so that the recordedcontent may be eventually displayed without disrupting the continuity ofthe program.

A query finds out if the fast forward feature has been implemented asindicated in diamond 1224. If so, a fast playback is implemented asshown in block 1226. Once the playback equals the current broadcast orprogram content (diamond 1228), the flow returns to display currentbroadcast or program content. The fast forward may progress at 300 to400% (or some other speed) of normal playback speed.

Moving on to FIG. 12A, at diamond 1230 a check is done to see if the 10second replay feature has been activated. If so, the storage device isoperated to find the content which was recorded 10 seconds ago (block1232). The recorded content is then played (block 1234) from the 10second period back to present. This would correspond to the replayfeature implemented with broadcast television sporting events. After the“rewind” and playback of 10 seconds has been completed, the system isautomatically transferred to automatic play at the content that wasrecorded while the replay is being displayed so the user does not loseany continuity in the show. Of course, the user can set the amount oftime for the replay to be something other than 10 seconds.

A catchup feature is checked (diamond 1236) and, if this feature hasbeen selected, an accelerated playback option is implemented (asindicated in block 1238). In the accelerated playback, the playback maybe slightly increased in speed, for example from 105 to 115% of actualspeed, so the user can easily follow the program but can make up forlost time to get back into the program as broadcast. Once the recordedprogram catches up to the real-time program as indicated in diamond1240, the catchup feature is completed and the user returns to real-timedisplay. A conventional technique may be used to maintain audio pitchduring accelerated playback.

Control over the record and playback functions may be implementedthrough a normal mouse function, for example, by clicking on icons asindicated in FIG. 11. In addition, the record and playback features maybe controlled remotely even when the user is not proximate to a keyboardor mouse. This may be done using a conventional remote control operatorwhich may, for example, utilize infrared radiation.

As indicated in the depiction of an example of a system 1000 (FIG. 10),an infrared adapter 1040 may be provided, for example, in compliancewith the standards and specifications (such as Infrared Data AssociationSerial Infrared Link Access Protocol Version 1.0, Jun. 23, 1994) of theInfrared Data Association (which can be found at {www.irda.org}). Theremote control 1300, as shown in FIG. 13, may be utilized as a separatestand alone remote control or its features may be incorporated into aglobal remote control. The illustrated control 1300 is a stand alonecontrol for controlling the record and broadcast features. It includesan infrared transmission port 1302 and a plurality of buttons to controlthe various features. For example, a zoom function 1312 may beimplemented as a four-way rocker switch. A zoom is implemented for thetop quadrant of a display by pressing the upper edge of the button. Ifthe button is pressed in the middle, the center quadrant is chosen forzoom. Likewise any of the remaining three edges of the zoom button canbe operated to select a desired quadrant for zoom features.

A button 1306 is provided to implement the replay function, a button1308 may implement the pause feature, and an enter button 1310 may beprovided to allow entry of various information including a particulartelevision program station. An on/off button 1314 may also be provided.Fast forward may be implemented by button 1316, an electronic programguide may be called by pressing button 1304. The catch up feature may becalled by the button 1318, and a 10 second reply may be implemented bythe button 1320. A conventional numeric keyboard 1322 may be provided toinput additional information including selected television channels.

Techniques for storing the video stream onto the storage device 1410 andfor reading the information out of the storage device are summarized inTable 1 below and illustrated in FIGS. 14, 15, and 16. In FIG. 14, aschematic depiction of the storage system 1400 includes a digitalstorage device such as a hard disk drive 1410. The digitized video 1402is initially stored in a buffer which is designated as being currentlyon the top of the memory stack. The transfer of information between thebuffers and the storage device 1410 may be done in discrete time periodsreferred to herein as time steps.

TABLE 1 Time Steps Buffers 1 2 3 4 5 6 7 8 9 Input 1404 1408 1406 14041408 1406 1404 1408 1406 Storage Read X X X 1412 X 1414 X 1412 X 14141416 1416 Write X X 1404 X 1404 X 1406 X 1404 1408 1406 1408 1408 OutputX X X X 1414 1412 1416 1414 1412 (X = no action)In a first time step, shown in FIG. 15, the digitized video 1402 (FIG.14) is stored in memory buffer 1404 because that buffer is currently atthe top of the memory stack, as indicated in block 1502 in FIG. 15.

As the buffer 1404 fills up, the buffer 1408 moves to the top of thestack (as indicated by the dashed arrow) and incoming video is stored inbuffer 1408. As indicated in block 1504 in FIG. 15 in time step 2, thebuffer 1408 replaces the buffer 1404 as the designated top of the stackbuffer. The next input video is then stored in the new buffer (1408) asindicated in block 1506.

In time step 3 the buffer 1408 has filled up and the contents of buffers1404 and 1408 are written to the storage device 1410 in a single writeoperation. During the same time step, buffer 1406 moves to the top ofthe stack and becomes the storage buffer for incoming video. This isillustrated in blocks 1508, 1510 and 1512 in FIG. 15.

In time step 4, the buffer 1404 moves back to the top of the stack tostore incoming video since its previous content has already been savedin the storage device 1410. This is indicated in block 1514 of FIG. 15.Additionally, in time step 4, the content of buffer 1408 is written tothe storage device 1410 as illustrated in block 1515. The storing ofincoming information, as illustrated in FIG. 15 then cycles back to thetop of the flow in FIG. 15 and continues in the same fashion thereafter.

The content from the storage device 1410 is then read into buffers 1412and 1414 in time step 4 and block 1602.

In time step 5, the buffer 1408 moves to the top of the stack to storeincoming video, and buffer 1414 moves to the top of the output stack andtransfers data to the video output 1418. The contents of the buffers1404 and 1406 are then written to the storage device 1410.

The time steps 1 to 5 illustrate a complete cycle from input to output.The remaining sequence of steps repeat starting at step 1 through 4 forthe input of data and time steps 4 through 9 repeat for the output ofdata.

Referring now solely to FIGS. 14 and 16, in time step 6, informationstored in the storage device is read to the sixth and fourth buffers(i.e., buffers 1414 and 1416) as indicated in block 1606. The contentsof the fifth buffer (1412) are sent to the output port 1418.

In time step 7, the contents of the sixth buffer (which is buffer 1416)are sent to the output port 1418. No other output operations occur.

In time step 8, the contents from the storage device 1410 are read intothe fifth and sixth buffers (buffers 1412 and 1416) as indicated inblock 1612. Also the contents of the fourth buffer (buffer 1414) arepassed to the output port 1418 as indicated in block 1614.

Finally, during time step 9 the contents of the fifth buffer (buffer1412) are passed to the output port 1418, completing the cycle.

Using these techniques, the storage device is provided with enough time,through the operation of the buffers, to write the incoming video whilesupplying enough data simultaneously to the output display. Since thememory is used to make sure no content is dropped at the input end andthe display is never starved at the output end, continuous display canbe provided without losing information. In addition reading and writinglarger chunks of data at one time reduces the amount of storage devicehead movement, thereby allowing the storage device to keep up with theread and write requests.

The size of the individual buffers 1404 to 1408 and 1412 to 1416 (“B”)in megabytes is dependent at least in part on a number of factorsincluding the input/output data rate “D”, for example in megabytes persecond, the effective bandwidth of the storage device when reading orwriting “S”, for example in megabytes per second, and the average seektime for the storage device per transaction “t”, in microseconds. Thetime that it takes to fill up one memory buffer (B divided by D) isadvantageously greater than the time it takes to read or write twomemory buffers (2B divided by S) plus the average seek time (t):

$( \frac{B}{D} ) \geq {( \frac{2B}{S} ) + ( \frac{t}{1000000} )}$

Solving for the minimum buffer size in the above equation yields thefollowing equation which gives the minimum buffer size:

$B \geq {( \frac{DS}{S - {2D}} ) \times ( \frac{t}{1000000} )}$

A viewer can pause a season premier of a television while the station isstill broadcasting in order to get up and answer the telephone. The usercan resume watching the show after interruption as if the program wereon video tape. In addition, while watching a sports event the user canrewind back to a controversial pass, replay the play in slow motion,stop at the exact instance when the catch was made and zoom in on thereceiver. Also, the user can apply an image sharpening filter to see ifboth of the receiver's feet touched the ground before the receiver wentout of bounds.

Timeshifting by substantially simultaneously recording and playing backa data stream enables the following type of benefits/features for anend-user. While watching the season premiere of a television show, theviewer can literally “pause” the program in order to get up and answerthe phone or get a snack from the refrigerator. After the interruption,the viewer can resume watching again without having missed anything. Ifthe viewer came home 15 minutes late for the show but had startedrecording the program from the beginning, the viewer can begin watchingright away instead of waiting for the show to be over and then rewindingthe cassette tape. Also the viewer can replay scenes during a “live”broadcast (e.g. season premiere of a show or sporting event) in case theviewer missed a line or an exciting play. In addition, while watching asports event the user can rewind back to a controversial play, replaythe play in slow motion, and stop at the exact instance when the catchwas made. To get a better view, the viewer can zoom in on a portion ofthe screen and apply image sharpening filters to magnify one part of thescreen.

An embodiment of a video transmission system 1800, shown in FIG. 18,includes a video transmitter 1802 which transmits video and associatedaudio to a plurality of receivers 1804. The video transmitter maytransmit video content using a variety of transport media includingtelevision broadcast, cable distribution, satellite distribution, andcomputer networks. The scene change detector 1806 also receives thevideo transmission from the transmitter 1802. The scene change detector1806 may be resident in the receiver 1804 or it may be separate entity,at a separate location, which provides scene change detection servicesfor a number of different receivers 1804. In any case, the scene changedetector 1806 may communicate in a two-way communication with thereceiver 1804.

The receiver 1804 may request scene change detection information fromthe scene change detector 1806. A shot is what is captured between arecord and stop camera operation. A scene is a sequence of contiguousshots unified by a common location or event. The technology forimplementing a scene change detector is well known in the industry. Aconventional scene change detector 1806 may be utilized or, if desired,a modified scene change detector may be implemented in accordance withan embodiment of the present invention described hereinafter.

The received video information, stored on the receiver 1804, may havethe file 1900 format shown in FIG. 19. Namely, the video information1904 may be stored in a file 1900 together with time tag information1906 and a channel identifier 1902. Thus, the video information isidentifiable both by the channel identifier of the channel on which itwas received and the local time when it was received, in one embodimentof the present invention.

The receiver 1804 may request scene change information from the scenechange detector 1806 by making a request in the format illustrated inFIG. 20 in one embodiment of the present invention. The request 2000 mayinclude an identifier 2002 for the channel on which the desired scenechange information is provided, information about the time of theprogram 2004 and an identifier 2006 to identify the requesting receiver.

The scene change detector 1806 may respond, for example, by sending afile 2100 in the format, shown in FIG. 21, including an identifier 2102to identify the appropriate receiver 1804 which made the request forscene change information. A channel code 2104 may be included togetherwith a plurality of scene change times 2106.

Thus, for a given channel and program, the scene change detector 1806may provide a very compact amount of information sufficient to locatescene changes in the received video. In particular, the scene changedetector 1806 may provide the times in minutes and seconds at which eachscene change occurs. The receiver 1804 may then use this information toretrieve the particular scenes. For example, the receiver 1804 maycreate and display a storyboard including a frame representative of eachdetected scene so that a particular scene may be identified.

Referring next to FIG. 22, the scene change detection software 2200,stored in the receiver 1804, may initially request scene change times,as indicated in block 2202. The request may be in the format indicatedin FIG. 20, for example. The scene change times are received from thescene change detector as indicated in block 2204. The response may be inthe format indicated in FIG. 21, for example. The time information 2106is scanned and used to collect the corresponding stored video files, asindicated in block 2206. The files may be in the format indicated inFIG. 19 in one embodiment of the present invention. For example, thetime tags 1906 may be matched to the scene change times 2106.

The various scenes associated with the files 1900 may be displayed forexample in a storyboard fashion, as indicated in block 2208.Alternatively, a plurality of sequential scenes may be displayed and theuser may scroll the display to see prior or subsequent scenes.

Referring now to FIG. 23, the scene change analyzer which may beutilized to implement the scene change detector 1806 in accordance withone embodiment of the present invention may use the software 2300.Initially a timer may be started, as indicated in block 2302. Then acheck at diamond 2306 determines whether a particular scene can beidentified using conventional techniques. If so, the scene ifbookmarked, and the bookmark is stored as indicated in block 2316. Ifnot, an analysis of the audio track may be utilized to identify scenechange information, as indicated in diamond 2308.

For example, a change in the subject matter of the audio dialog may beutilized to identify scene changes. In one embodiment of the presentinvention, in a conventional news program, the same announcer may bepresent in a number of frames. Using video analysis alone, it may bedifficult to identify a scene change. However, by analyzing the audioinformation, it may be possible to determine when a subject matterchange has occurred. This audio analysis may be done by identifyingchanges in tone, pace and quiet times.

Alternatively, an analysis of a closed caption script may be implementedat diamond 2310. The actual words utilized in the ongoing dialog may beanalyzed to identify that a new set of words are being utilized. Thisinformation may be then used to identify a scene change.

Finally, if none of the other techniques have been successful, digitalvideo compression data may be analyzed to locate scene changes. Inconventional compression algorithms such as the Moving Pictures ExpertsGroup (MPEG-2) compression algorithm, one can analyze each of the framesto determine where scene changes occur. For example, I-frames or “intra”frames are coded as stand-alone still images. They generally are used atscene cuts. However, P and B frames may also provide useful information.MPEG-2 refers to International Organization for Standardization (ISO)and International Electrotechnical Commission (IEC) Standard 13818-1(Systems), 13818-2 (video), 13818-3 (audio) and 13818-4 (compliance)(1994). Normally, compression algorithms compress data, such as videodata, by finding scene changes and using that information to compressthe information which must be stored. Information about discontinuitiesin the spatial domain may be extracted and used to identify scenechanges in an expeditious fashion for display purposes. See B. L. Yeoand B. Liu, “Rapid Scene Analysis on Compressed Video,” IEEE Trans. onCircuits for Video Technology, Vol. 5, No. 6, December 1995.

If none of the techniques is successful after a sufficient number oftries, the timer eventually expires, as indicated at diamond 2314. Ashot is then chosen, based on the passage of time, as a scene change anda bookmark is applied to that scene. Thus, if the algorithm is unable toidentify a scene change, after a sufficient time has passed, a shot issimply chosen as a representative scene change and the flow ends.

In some cases, users may desire to have a large number of representativescenes. Thus, they may desire a relatively high scene change detectionsensitivity. Other users may be less desirous of such a detailed reportand may prefer a lower sensitivity. FIG. 24 shows a graphical userinterface 2400 which may be utilized to allow the user to inputinformation into the receiver 1804 about the desired scene changesensitivity in one embodiment of the present invention. The user canimplement a scene change rating at the data entry point 2402, or asliding scale 2404 may be adjusted to indicate the level of sensitivitythat is desired. The level of sensitivity may be entered to request moreor less selectivity in identifying scene changes or to increase the timeof the timer used in block 2302 in FIG. 23.

The video information received by the receiver 1804 may be in analog ordigital formats in accordance with modem video transmission protocols.For example, digital format television broadcasts are available under avariety of different digital standard including the Advanced TelevisionSystems Committee (ATSC) Digital Television Standard, issued on Sep. 15,1995, as one example. Thus, it is advantageous to detect whetherincoming video is in one of the digital formats. For example, if theinformation is already in digital format, compression is not necessarybefore storage. However, if the information is in an analog format, itmay be desirable to convert the information to a digital format and thento compress the information.

One technique for handling dual mode (analog or digital) data, uses thesoftware 2500 illustrated in FIG. 25. Incoming video is received asindicated in block 2502. An attempt is made to demodulate the incomingvideo as indicated in block 2504. If the information uses vertical sideband modulation (VSB) with 8 (terrestrial) or 16 (high data rate)discrete amplitude levels (8 VSB and 16 VSB) it is a digitaltransmission and can be demodulated. If this attempt is successful, asdetermined at diamond 2506, it can be assumed safely that the data isdigital. In such case, it is not necessary to encode and compress theinformation. If the attempt is unsuccessful, the information may beassumed to be analog, and in such case, the information an be convertedto a digital format and compressed as indicated in block 2508.Regardless of its format, the incoming data is subjected to a scenechange detection analysis as indicated in block 2510 and then stored asindicated at 2512.

In some digital broadcast protocols, such as the ATSC standard citedearlier, more than one video program may be transmitted in a givenchannel. Thus, referring to FIG. 26, a channel 2602 may include up tofour programs 2604. Each program may have associated with it a programidentifier 2606. This enables selection of one program over another.

Thus, in one embodiment of the present invention, it is possible to jumpbetween viewing any of the programs in a given channel since all fourprograms may be stored automatically. Referring to FIG. 27, the software2700 initially determines whether a given transmission is a digitaltransmission (diamond 2702). If not, it can be assumed that there isonly one program per channel. Otherwise, a determination is made adiamond 2704 whether there are actually multiple programs in a givenchannel. If so, a check at diamond 2706 determines whether the user hasindicated a desire to implement a program switch from one program toanother. A program switch may be entered using a graphical userinterface as one example. When a program switch command is received, theswitch time may be stored as indicated in block 2708. Next, the programis switched within the same channel by feeding the appropriate storedinformation such as the program identifier 2606 of the desired program2604.

When the user wishes to resume to the prior program, as indicated atdiamond 2712, the restoration, implemented at block 2714, uses thestored switch time to identify a return point in the prior program. Inthis way, the user can watch one program in the same channel, switch toanother program, return to the original program and, using the catch-upfeature, may catch back up to real time, by knowing where the viewerleft the prior program. The user can jump, for example using a remotecontrol unit, between the various programs even within the same channel.

Where analog or digital video information is received, it is desirableto avoid to the greatest possible extent, decompressing the receivedvideo in order to detect scene changes and recompressing the video inorder to store it in accordance with the principles described herein. Byusing the so-called DC images, it is possible to extract sufficientinformation to determine where shots or scene changes occur withoutcompletely decompressing the video.

DC images are spatially reduced versions of the original images. Forexample, referring to FIG. 28, the original image 2800 in digital formatmay have data corresponding to eight blocks 2802. The DC image 2804 mayhave spatially reduced versions 2806 of the same eight blocks. Generallyan image may be divided into blocks of N by N pixels. The (i, j) pixelof the DC image is the average value of the (i, j) block of the originalimage. Sequences formed in such a manner may be called DC sequences.

For example, an original image of 320 by 240 pixels may have a DC imageof 40 by 30 pixels using N equal 8 as illustrated in FIG. 28.

The smaller data size of the DC image makes it more reasonable toprocess every frame in a video sequence. The DC sequence may then beprocessed, for example in one embodiment of the present invention, usingthree different detection algorithms, one which detects abrupt changes,one that detects plateaus and one that detects flashlights. See B. L.Yeo and B. Liu, “Rapid Scene Analysis on Compressed Video”, IEEE Trans.on Circuits and Systems for Video Technology, Vol. 5, No. 6, December1995.

The DC image is derived from a discrete cosign transform (DCT) based andmotion compensated DCT-based compression techniques such as motion JPEGand MPEG. The same principles of first extracting only reducedresolution images and then using said reduced images for processingpurposes applies to many other compression formats including those ofRadius, Cinepak, Intel's Indeo, subband/wavelet based compression andthe like.

By using DC images and DC sequences for shot or scene change analysis,the operations are performed directly on compressed data, eliminatingthe need for full frame decompression. In addition, because only a smallfraction of the original data is used, the computational complexity maybe reduced. See the previously cited Yeo and Liu paper. In someinstances, the P and B frames alone may yield DC images that are veryclose to the actual images. The computational cost per DC coefficient inP and B frames is reduced to four multiplications.

Referring to FIG. 29, a software flow 2900 for extracting theinformation needed to detect shots or scene changes in compresseddigital video begins by receiving the compressed video as indicated inblock 2902. The DC image is extracted (block 2904) and the shot or scenechange information may be determined from any combination of P, B orI-frame information as explained in the cited Yeo paper. The compressedvideo is then stored as described in connection with the presentinvention.

Because there is no need to decompress the data, the speed of storagemay be greatly increased and the computational complexity may bereduced. This facilitates a system which substantially simultaneouslyrecords and plays back a data stream, of the type described herein,since large amounts of data may be stored at one time and the resourcesof the system may be taxed in undertaking this process. The additionalburden of extracting the scene change information may tend to slow thisprocess down. By simplifying the shot or scene change detection processthrough the use of DC images, the computational complexity and thus thespeed may be improved.

Thus, an embodiment of a method of substantially simultaneouslyrecording and playing back a time-shifted video stream in accordancewith the invention is disclosed. The specific arrangements and methodsdescribed herein are merely illustrative of the principles of thisinvention. For example, a similar embodiment may be used to store andretrieve other types of data streams besides video streams. Numerousmodifications in form and detail may be made without departing from thescope of the described invention. Although this invention has been shownin relation to a particular embodiment, it should not be considered solimited. Rather, the described invention is limited only by the scope ofthe appended claims.

1. A method of detecting shots or scene changes in compressed videoinfoririation comprising: identifying a shot or scene change informationdeveloped by the algorithm used to compress said video information;using said shots or scene change information to identify a shot or scenechange; marking the location of said shot or scene change to facilitateidentifying the video frame where the shot or scene change occursincluding comparing the location of said shot or scene change toinformation stored in association with said video frames indicative ofthe time when the video frames were received; and storing saidcompressed video in a randomly accessible memory.
 2. The method of claim1 wherin identifying shot or scene change information includesidentifying I-frames developed by a compression algorithm.
 3. The methodof claim 1 further including displaying a plurality of framesrepresentative of the scene changes which have been identified.